{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e777354",
   "metadata": {},
   "source": [
    "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/pixeltable/pixeltable/blob/master/docs/release/tutorials/working-with-openai.ipynb)&nbsp;&nbsp;\n",
    "[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/tutorials/working-with-openai.ipynb)\n",
    "\n",
    "# Working with OpenAI in Pixeltable\n",
    "\n",
    "Pixeltable unifies data and computation into a table interface. In the [Pixeltable Basics](https://pixeltable.github.io/pixeltable/tutorials/pixeltable-basics/) tutorial, we saw how OpenAI API calls can be incorporated into Pixeltable workflows. In this tutorial, we'll go into more depth on OpenAI integration. You'll need an OpenAI API key to run this demo.\n",
    "\n",
    "### Prerequisites\n",
    "- An OpenAI account with an API key (https://openai.com/index/openai-api/)\n",
    "\n",
    "### Important Notes\n",
    "\n",
    "- OpenAI usage may incur costs based on your OpenAI plan.\n",
    "- Be mindful of sensitive data and consider security measures when integrating with external services."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b2e6912-e936-4c3a-84a2-ba99950c9493",
   "metadata": {},
   "source": [
    "First you'll need to install required libraries and enter your OpenAI API key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5288926-c278-4cbc-815c-cbc0433bbf49",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q pixeltable openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "385f6831-f029-42bb-99f1-652a809ffc6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter your OpenAI API key: ········\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import getpass\n",
    "if 'OPENAI_API_KEY' not in os.environ:\n",
    "    os.environ['OPENAI_API_KEY'] = getpass.getpass('Enter your OpenAI API key:')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d3dd131-22de-496c-9f02-ffd4515c20d3",
   "metadata": {},
   "source": [
    "Now let's create a Pixeltable directory to hold the tables for our demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9bdc613f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected to Pixeltable database at: postgresql://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata\n",
      "Created directory `demo`.\n"
     ]
    }
   ],
   "source": [
    "import pixeltable as pxt\n",
    "\n",
    "pxt.drop_dir('demo', force=True)  # Ensure a clean slate for the demo\n",
    "pxt.create_dir('demo')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa71d21e",
   "metadata": {},
   "source": [
    "## Creating the Table\n",
    "\n",
    "First, we'll create a table and populate it with some sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a6a7bd0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created table `openai`.\n"
     ]
    }
   ],
   "source": [
    "t = pxt.create_table('demo.openai', {'id': pxt.IntType(), 'input': pxt.StringType()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "342407c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# text from https://en.wikipedia.org/wiki/Global_financial_crisis_in_September_2008\n",
    "\n",
    "wikipedia_text = '''On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.\n",
    "The significance of the Lehman Brothers bankruptcy is disputed with some assigning it a pivotal role in the unfolding of subsequent events.\n",
    "The principals involved, Ben Bernanke and Henry Paulson, dispute this view, citing a volume of toxic assets at Lehman which made a rescue impossible.[16][17] Immediately following the bankruptcy, JPMorgan Chase provided the broker dealer unit of Lehman Brothers with $138 billion to \"settle securities transactions with customers of Lehman and its clearance parties\" according to a statement made in a New York City Bankruptcy court filing.[18]\n",
    "The same day, the sale of Merrill Lynch to Bank of America was announced.[19] The beginning of the week was marked by extreme instability in global stock markets, with dramatic drops in market values on Monday, September 15, and Wednesday, September 17.\n",
    "On September 16, the large insurer American International Group (AIG), a significant participant in the credit default swaps markets, suffered a liquidity crisis following the downgrade of its credit rating.\n",
    "The Federal Reserve, at AIG's request, and after AIG had shown that it could not find lenders willing to save it from insolvency, created a credit facility for up to US$85 billion in exchange for a 79.9% equity interest, and the right to suspend dividends to previously issued common and preferred stock.[20]'''\n",
    "\n",
    "sample_inputs = wikipedia_text.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c5f0b862",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells:   0%|                                                    | 0/1 [00:00<?, ? cells/s]\n",
      "Inserting rows into `openai`: 1 rows [00:00, 491.37 rows/s]\n",
      "Computing cells: 100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 209.30 cells/s]\n",
      "Inserted 1 row with 0 errors.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>id</th>\n",
       "      <th>input</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "   id                                              input\n",
       "0   0  On Sunday, September 14, it was announced that..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Insert a single sample row into the table\n",
    "t.insert(id=0, input=sample_inputs[0])\n",
    "t.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b08ecc3",
   "metadata": {},
   "source": [
    "## Making OpenAI API calls\n",
    "\n",
    "Calling OpenAI API endpoints involves constructing a message object, which we can express in Pixeltable by adding a new computed column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "12197f4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 171.38 cells/s]\n",
      "Added 1 column value with 0 errors.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "UpdateStatus(num_rows=1, num_computed_values=1, num_excs=0, updated_cols=[], cols_with_excs=[])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt = \"For the following sentence, extract all company names from the text.\"\n",
    "\n",
    "msgs = [\n",
    "    { \"role\": \"system\", \"content\": prompt },\n",
    "    { \"role\": \"user\", \"content\": t.input }\n",
    "]\n",
    "\n",
    "t.add_column(input_msgs=msgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92068113",
   "metadata": {},
   "source": [
    "Unlike the values of the`input` column, which users provide, the `t.input_msgs` column is computed automatically from the `t.input` column values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "249592fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>id</th>\n",
       "      <th>input</th>\n",
       "      <th>input_msgs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;For the following sentence, extract all company names from the text.&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.&quot;}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "   id                                              input  \\\n",
       "0   0  On Sunday, September 14, it was announced that...   \n",
       "\n",
       "                                          input_msgs  \n",
       "0  [{'role': 'system', 'content': 'For the follow...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cea0186",
   "metadata": {},
   "source": [
    "In Pixeltable, OpenAI API calls are exposed as Pixeltable functions, which can be used to create computed columns. We can run the prompt against our input data using OpenAI's `chat_completions` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d0aaed0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|████████████████████████████████████████████| 1/1 [00:00<00:00,  1.48 cells/s]\n",
      "Added 1 column value with 0 errors.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>id</th>\n",
       "      <th>input</th>\n",
       "      <th>input_msgs</th>\n",
       "      <th>chat_output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;For the following sentence, extract all company names from the text.&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.&quot;}]</td>\n",
       "      <td>{&quot;id&quot;: &quot;chatcmpl-9mQUTc5fbQLwhYseYzjb1GpLLRBbB&quot;, &quot;model&quot;: &quot;gpt-4o-mini-2024-07-18&quot;, &quot;usage&quot;: {&quot;total_tokens&quot;: 68, &quot;prompt_tokens&quot;: 61, &quot;completion_tokens&quot;: 7}, &quot;object&quot;: &quot;chat.completion&quot;, &quot;choices&quot;: [{&quot;index&quot;: 0, &quot;message&quot;: {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Lehman Brothers, Federal Reserve Bank&quot;, &quot;tool_calls&quot;: null, &quot;function_call&quot;: null}, &quot;logprobs&quot;: null, &quot;finish_reason&quot;: &quot;stop&quot;}], &quot;created&quot;: 1721328601, &quot;system_fingerprint&quot;: &quot;fp_661538dc1f&quot;}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "   id                                              input  \\\n",
       "0   0  On Sunday, September 14, it was announced that...   \n",
       "\n",
       "                                          input_msgs  \\\n",
       "0  [{'role': 'system', 'content': 'For the follow...   \n",
       "\n",
       "                                         chat_output  \n",
       "0  {'id': 'chatcmpl-9mQUTc5fbQLwhYseYzjb1GpLLRBbB...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pixeltable.functions import openai\n",
    "\n",
    "t['chat_output'] = openai.chat_completions(model='gpt-4o-mini', messages=t.input_msgs)\n",
    "t.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004723a6",
   "metadata": {},
   "source": [
    "The output of the OpenAI API calls are generally complex JSON structures, which require some navigation to extract the response. We can express this as JSON path expressions and create another computed column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "06e15905",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 185.25 cells/s]\n",
      "Added 1 column value with 0 errors.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>id</th>\n",
       "      <th>input</th>\n",
       "      <th>input_msgs</th>\n",
       "      <th>chat_output</th>\n",
       "      <th>response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;For the following sentence, extract all company names from the text.&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.&quot;}]</td>\n",
       "      <td>{&quot;id&quot;: &quot;chatcmpl-9mQUTc5fbQLwhYseYzjb1GpLLRBbB&quot;, &quot;model&quot;: &quot;gpt-4o-mini-2024-07-18&quot;, &quot;usage&quot;: {&quot;total_tokens&quot;: 68, &quot;prompt_tokens&quot;: 61, &quot;completion_tokens&quot;: 7}, &quot;object&quot;: &quot;chat.completion&quot;, &quot;choices&quot;: [{&quot;index&quot;: 0, &quot;message&quot;: {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Lehman Brothers, Federal Reserve Bank&quot;, &quot;tool_calls&quot;: null, &quot;function_call&quot;: null}, &quot;logprobs&quot;: null, &quot;finish_reason&quot;: &quot;stop&quot;}], &quot;created&quot;: 1721328601, &quot;system_fingerprint&quot;: &quot;fp_661538dc1f&quot;}</td>\n",
       "      <td>Lehman Brothers, Federal Reserve Bank</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "   id                                              input  \\\n",
       "0   0  On Sunday, September 14, it was announced that...   \n",
       "\n",
       "                                          input_msgs  \\\n",
       "0  [{'role': 'system', 'content': 'For the follow...   \n",
       "\n",
       "                                         chat_output  \\\n",
       "0  {'id': 'chatcmpl-9mQUTc5fbQLwhYseYzjb1GpLLRBbB...   \n",
       "\n",
       "                                response  \n",
       "0  Lehman Brothers, Federal Reserve Bank  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t['response'] = t.chat_output.choices[0].message.content\n",
    "t.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96d19ca5",
   "metadata": {},
   "source": [
    "Let's run a query to look only at the input and output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4313dbb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>input</th>\n",
       "      <th>response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "      <td>Lehman Brothers, Federal Reserve Bank</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "                                               input  \\\n",
       "0  On Sunday, September 14, it was announced that...   \n",
       "\n",
       "                                response  \n",
       "0  Lehman Brothers, Federal Reserve Bank  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.select(t.input, t.response).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ae2854d",
   "metadata": {},
   "source": [
    "Once we have defined these computed columns, much like with a spreadsheet, newly inserted `t.input` values trigger computation of all derived columns, such as `t.response`. Let's insert the rest of our data and see how this works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d5144a35",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|██████████████████████████████████████████| 20/20 [00:02<00:00,  8.20 cells/s]\n",
      "Inserting rows into `openai`: 5 rows [00:00, 761.63 rows/s]\n",
      "Computing cells: 100%|██████████████████████████████████████████| 20/20 [00:02<00:00,  8.16 cells/s]\n",
      "Inserted 5 rows with 0 errors.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "UpdateStatus(num_rows=5, num_computed_values=20, num_excs=0, updated_cols=[], cols_with_excs=[])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.insert({'id': i, 'input': sample_inputs[i]} for i in range(1, len(sample_inputs)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "729713aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>input</th>\n",
       "      <th>response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "      <td>Lehman Brothers, Federal Reserve Bank</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The significance of the Lehman Brothers bankruptcy is disputed with some assigning it a pivotal role in the unfolding of subsequent events.</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The principals involved, Ben Bernanke and Henry Paulson, dispute this view, citing a volume of toxic assets at Lehman which made a rescue impossible.[16][17] Immediately following the bankruptcy, JPMorgan Chase provided the broker dealer unit of Lehman Brothers with \\$138 billion to &quot;settle securities transactions with customers of Lehman and its clearance parties&quot; according to a statement made in a New York City Bankruptcy court filing.[18]</td>\n",
       "      <td>The company names extracted from the text are:\n",
       "1. Lehman\n",
       "2. JPMorgan Chase\n",
       "3. Lehman Brothers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The same day, the sale of Merrill Lynch to Bank of America was announced.[19] The beginning of the week was marked by extreme instability in global stock markets, with dramatic drops in market values on Monday, September 15, and Wednesday, September 17.</td>\n",
       "      <td>Merrill Lynch, Bank of America</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>On September 16, the large insurer American International Group (AIG), a significant participant in the credit default swaps markets, suffered a liquidity crisis following the downgrade of its credit rating.</td>\n",
       "      <td>American International Group (AIG)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The Federal Reserve, at AIG&#x27;s request, and after AIG had shown that it could not find lenders willing to save it from insolvency, created a credit facility for up to US\\$85 billion in exchange for a 79.9% equity interest, and the right to suspend dividends to previously issued common and preferred stock.[20]</td>\n",
       "      <td>Federal Reserve, AIG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "                                               input  \\\n",
       "0  On Sunday, September 14, it was announced that...   \n",
       "1  The significance of the Lehman Brothers bankru...   \n",
       "2  The principals involved, Ben Bernanke and Henr...   \n",
       "3  The same day, the sale of Merrill Lynch to Ban...   \n",
       "4  On September 16, the large insurer American In...   \n",
       "5  The Federal Reserve, at AIG's request, and aft...   \n",
       "\n",
       "                                            response  \n",
       "0              Lehman Brothers, Federal Reserve Bank  \n",
       "1                                    Lehman Brothers  \n",
       "2  The company names extracted from the text are:...  \n",
       "3                     Merrill Lynch, Bank of America  \n",
       "4                 American International Group (AIG)  \n",
       "5                               Federal Reserve, AIG  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.select(t.input, t.response).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0cfe126",
   "metadata": {},
   "source": [
    "## Adding Ground Truth Data\n",
    "\n",
    "Now let's see how Pixeltable can be used to evaluate a model against ground truth data. We'll start by manually populating a `ground_truth` column in our table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6d4f8611",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|███████████████████████████████████████████| 6/6 [00:00<00:00, 675.99 cells/s]\n",
      "Added 6 column values with 0 errors.\n",
      "Inserting rows into `openai`: 1 rows [00:00, 988.76 rows/s]\n",
      "Inserting rows into `openai`: 1 rows [00:00, 1540.89 rows/s]\n",
      "Inserting rows into `openai`: 1 rows [00:00, 2020.38 rows/s]\n",
      "Inserting rows into `openai`: 1 rows [00:00, 1893.59 rows/s]\n",
      "Inserting rows into `openai`: 1 rows [00:00, 1312.36 rows/s]\n",
      "Inserting rows into `openai`: 1 rows [00:00, 2150.93 rows/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 1 rows [00:00, 2136.68 rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 0 rows [00:00, ? rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 1 rows [00:00, 2308.37 rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 0 rows [00:00, ? rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 1 rows [00:00, 2432.89 rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 0 rows [00:00, ? rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 1 rows [00:00, 2777.68 rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 0 rows [00:00, ? rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Inserting rows into `openai`: 1 rows [00:00, 2855.21 rows/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "t['ground_truth'] = pxt.StringType(nullable=True)\n",
    "\n",
    "ground_truth = [\n",
    "    'Lehman Brothers',\n",
    "    'Lehman Brothers',\n",
    "    'JP Morgan Chase, Lehman Brothers',\n",
    "    'Merill Lynch, Bank of America',\n",
    "    'American International Group',\n",
    "    'American International Group',\n",
    "]\n",
    "\n",
    "for i, gt in enumerate(ground_truth):\n",
    "    t.update({'ground_truth': gt}, where=(t.id == i))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17406f13",
   "metadata": {},
   "source": [
    "And this is what we have so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "72457011",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>input</th>\n",
       "      <th>response</th>\n",
       "      <th>ground_truth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>On Sunday, September 14, it was announced that Lehman Brothers would file for bankruptcy after the Federal Reserve Bank declined to participate in creating a financial support facility for Lehman Brothers.</td>\n",
       "      <td>Lehman Brothers, Federal Reserve Bank</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The significance of the Lehman Brothers bankruptcy is disputed with some assigning it a pivotal role in the unfolding of subsequent events.</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The principals involved, Ben Bernanke and Henry Paulson, dispute this view, citing a volume of toxic assets at Lehman which made a rescue impossible.[16][17] Immediately following the bankruptcy, JPMorgan Chase provided the broker dealer unit of Lehman Brothers with \\$138 billion to &quot;settle securities transactions with customers of Lehman and its clearance parties&quot; according to a statement made in a New York City Bankruptcy court filing.[18]</td>\n",
       "      <td>The company names extracted from the text are:\n",
       "1. Lehman\n",
       "2. JPMorgan Chase\n",
       "3. Lehman Brothers</td>\n",
       "      <td>JP Morgan Chase, Lehman Brothers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The same day, the sale of Merrill Lynch to Bank of America was announced.[19] The beginning of the week was marked by extreme instability in global stock markets, with dramatic drops in market values on Monday, September 15, and Wednesday, September 17.</td>\n",
       "      <td>Merrill Lynch, Bank of America</td>\n",
       "      <td>Merill Lynch, Bank of America</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>On September 16, the large insurer American International Group (AIG), a significant participant in the credit default swaps markets, suffered a liquidity crisis following the downgrade of its credit rating.</td>\n",
       "      <td>American International Group (AIG)</td>\n",
       "      <td>American International Group</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The Federal Reserve, at AIG&#x27;s request, and after AIG had shown that it could not find lenders willing to save it from insolvency, created a credit facility for up to US\\$85 billion in exchange for a 79.9% equity interest, and the right to suspend dividends to previously issued common and preferred stock.[20]</td>\n",
       "      <td>Federal Reserve, AIG</td>\n",
       "      <td>American International Group</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "                                               input  \\\n",
       "0  On Sunday, September 14, it was announced that...   \n",
       "1  The significance of the Lehman Brothers bankru...   \n",
       "2  The principals involved, Ben Bernanke and Henr...   \n",
       "3  The same day, the sale of Merrill Lynch to Ban...   \n",
       "4  On September 16, the large insurer American In...   \n",
       "5  The Federal Reserve, at AIG's request, and aft...   \n",
       "\n",
       "                                            response  \\\n",
       "0              Lehman Brothers, Federal Reserve Bank   \n",
       "1                                    Lehman Brothers   \n",
       "2  The company names extracted from the text are:...   \n",
       "3                     Merrill Lynch, Bank of America   \n",
       "4                 American International Group (AIG)   \n",
       "5                               Federal Reserve, AIG   \n",
       "\n",
       "                       ground_truth  \n",
       "0                   Lehman Brothers  \n",
       "1                   Lehman Brothers  \n",
       "2  JP Morgan Chase, Lehman Brothers  \n",
       "3     Merill Lynch, Bank of America  \n",
       "4      American International Group  \n",
       "5      American International Group  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.select(t.input, t.response, t.ground_truth).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce291c35",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Now that we have some ground truth available, we can carry out basic evaluations of the GPT outputs, in this case by asking ChatGPT to decide whether the two are equivalent. \n",
    "\n",
    "To start with, we'll create an evaluation prompt. In this case, the prompt requires some bespoke string substitution, so it's easiest to do using a UDF. (See the [Pixeltable Basics](https://pixeltable.github.io/pixeltable/tutorials/pixeltable-basics/) tutorial and the [UDFs in Pixeltable](https://pixeltable.github.io/pixeltable/howto/udfs-in-pixeltable/) guide for more details on UDFs.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c49ea5b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|███████████████████████████████████████████| 6/6 [00:00<00:00, 506.07 cells/s]\n",
      "Added 6 column values with 0 errors.\n"
     ]
    }
   ],
   "source": [
    "system_prompt = '''\n",
    "Compare the following listA and listB of entities, and check if they contain the same entities.\n",
    "Return a JSON object with the following format:\n",
    "{\"reasoning\": explaining your reasoning, \"decision\": 1 if the lists matched, 0 otherwise}\n",
    "'''\n",
    "\n",
    "@pxt.udf\n",
    "def eval_prompt(listA: str, listB: str) -> list[dict]:\n",
    "    return [\n",
    "        {'role': 'system', 'content': system_prompt},\n",
    "        {'role': 'user', 'content': f'listA: \"{listA}\" \\n listB: \"{listB}\"'}\n",
    "    ]\n",
    "\n",
    "t['eval_prompt'] = eval_prompt(t.response, t.ground_truth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "08445095",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>eval_prompt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;\\nCompare the following listA and listB of entities, and check if they contain the same entities.\\nReturn a JSON object with the following format:\\n{\\&quot;reasoning\\&quot;: explaining your reasoning, \\&quot;decision\\&quot;: 1 if the lists matched, 0 otherwise}\\n&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;listA: \\&quot;Lehman Brothers, Federal Reserve Bank\\&quot; \\n listB: \\&quot;Lehman Brothers\\&quot;&quot;}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;\\nCompare the following listA and listB of entities, and check if they contain the same entities.\\nReturn a JSON object with the following format:\\n{\\&quot;reasoning\\&quot;: explaining your reasoning, \\&quot;decision\\&quot;: 1 if the lists matched, 0 otherwise}\\n&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;listA: \\&quot;The company names extracted from the text are:\\n1. Lehman\\n2. JPMorgan Chase\\n3. Lehman Brothers\\&quot; \\n listB: \\&quot;JP Morgan Chase, Lehman Brothers\\&quot;&quot;}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;\\nCompare the following listA and listB of entities, and check if they contain the same entities.\\nReturn a JSON object with the following format:\\n{\\&quot;reasoning\\&quot;: explaining your reasoning, \\&quot;decision\\&quot;: 1 if the lists matched, 0 otherwise}\\n&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;listA: \\&quot;American International Group (AIG)\\&quot; \\n listB: \\&quot;American International Group\\&quot;&quot;}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;\\nCompare the following listA and listB of entities, and check if they contain the same entities.\\nReturn a JSON object with the following format:\\n{\\&quot;reasoning\\&quot;: explaining your reasoning, \\&quot;decision\\&quot;: 1 if the lists matched, 0 otherwise}\\n&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;listA: \\&quot;Lehman Brothers\\&quot; \\n listB: \\&quot;Lehman Brothers\\&quot;&quot;}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;\\nCompare the following listA and listB of entities, and check if they contain the same entities.\\nReturn a JSON object with the following format:\\n{\\&quot;reasoning\\&quot;: explaining your reasoning, \\&quot;decision\\&quot;: 1 if the lists matched, 0 otherwise}\\n&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;listA: \\&quot;Merrill Lynch, Bank of America\\&quot; \\n listB: \\&quot;Merill Lynch, Bank of America\\&quot;&quot;}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;\\nCompare the following listA and listB of entities, and check if they contain the same entities.\\nReturn a JSON object with the following format:\\n{\\&quot;reasoning\\&quot;: explaining your reasoning, \\&quot;decision\\&quot;: 1 if the lists matched, 0 otherwise}\\n&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;listA: \\&quot;Federal Reserve, AIG\\&quot; \\n listB: \\&quot;American International Group\\&quot;&quot;}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "                                         eval_prompt\n",
       "0  [{'role': 'system', 'content': '\n",
       "Compare the f...\n",
       "1  [{'role': 'system', 'content': '\n",
       "Compare the f...\n",
       "2  [{'role': 'system', 'content': '\n",
       "Compare the f...\n",
       "3  [{'role': 'system', 'content': '\n",
       "Compare the f...\n",
       "4  [{'role': 'system', 'content': '\n",
       "Compare the f...\n",
       "5  [{'role': 'system', 'content': '\n",
       "Compare the f..."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.select(t.eval_prompt).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8be46a",
   "metadata": {},
   "source": [
    "The actual evaluation happens in another computed column. We can use OpenAI's handy `response_format` parameter to enforce that the output is properly formed JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "43516050",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|████████████████████████████████████████████| 6/6 [00:06<00:00,  1.13s/ cells]\n",
      "Added 6 column values with 0 errors.\n",
      "Computing cells: 100%|███████████████████████████████████████████| 6/6 [00:00<00:00, 714.45 cells/s]\n",
      "Added 6 column values with 0 errors.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Added 6 column values with 0 errors.\n"
     ]
    }
   ],
   "source": [
    "t['eval'] = openai.chat_completions(model='gpt-4o-mini', messages=t.eval_prompt, response_format={'type': 'json_object'})\n",
    "t['eval_output'] = t.eval.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c72427a8",
   "metadata": {},
   "source": [
    "Let's take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5ce36880",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>response</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>eval_output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>American International Group (AIG)</td>\n",
       "      <td>American International Group</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;The first list contains &#x27;American International Group (AIG)&#x27; which includes the full name along with its acronym, while the second list contains only &#x27;American International Group&#x27; without the acronym. Although both represent the same entity, the presence of &#x27;(AIG)&#x27; in listA makes them differ slightly in terms of representation. However, they refer to the same organization.&quot;, &quot;decision&quot;: 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Federal Reserve, AIG</td>\n",
       "      <td>American International Group</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;The entities in listA are &#x27;Federal Reserve&#x27; and &#x27;AIG&#x27;. The entity &#x27;AIG&#x27; stands for &#x27;American International Group&#x27;, so both lists refer to the same organization. However, &#x27;Federal Reserve&#x27; is not mentioned in listB. Therefore, since listB does not contain all the entities from listA, the lists do not match.&quot;, &quot;decision&quot;: 0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Merrill Lynch, Bank of America</td>\n",
       "      <td>Merill Lynch, Bank of America</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;The first entity in listA is &#x27;Merrill Lynch&#x27; while in listB it is &#x27;Merill Lynch&#x27;. The correct spelling is &#x27;Merrill Lynch&#x27; with a double &#x27;r&#x27;. Since there is a spelling error in listB, the two lists do not contain the same entities.&quot;, &quot;decision&quot;: 0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The company names extracted from the text are:\n",
       "1. Lehman\n",
       "2. JPMorgan Chase\n",
       "3. Lehman Brothers</td>\n",
       "      <td>JP Morgan Chase, Lehman Brothers</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;ListA contains three entities: &#x27;Lehman&#x27;, &#x27;JPMorgan Chase&#x27;, and &#x27;Lehman Brothers&#x27;. ListB has two entities: &#x27;JP Morgan Chase&#x27; and &#x27;Lehman Brothers&#x27;. Although &#x27;Lehman Brothers&#x27; is common between both lists, &#x27;JPMorgan Chase&#x27; from ListA is written as &#x27;JP Morgan Chase&#x27; in ListB, which indicates a discrepancy in the naming format. Therefore, the two lists do not contain the exact same entities.&quot;, &quot;decision&quot;: 0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Lehman Brothers, Federal Reserve Bank</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;ListA contains &#x27;Lehman Brothers&#x27; and &#x27;Federal Reserve Bank&#x27;, while ListB contains only &#x27;Lehman Brothers&#x27;. Since ListB is missing &#x27;Federal Reserve Bank&#x27;, the two lists do not contain the same entities.&quot;, &quot;decision&quot;: 0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;Both lists contain the exact same entity &#x27;Lehman Brothers&#x27;. There are no differences in spelling, capitalization, or additional characters. Therefore, the lists match perfectly.&quot;, &quot;decision&quot;: 1}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "                                            response  \\\n",
       "0                 American International Group (AIG)   \n",
       "1                               Federal Reserve, AIG   \n",
       "2                     Merrill Lynch, Bank of America   \n",
       "3  The company names extracted from the text are:...   \n",
       "4              Lehman Brothers, Federal Reserve Bank   \n",
       "5                                    Lehman Brothers   \n",
       "\n",
       "                       ground_truth  \\\n",
       "0      American International Group   \n",
       "1      American International Group   \n",
       "2     Merill Lynch, Bank of America   \n",
       "3  JP Morgan Chase, Lehman Brothers   \n",
       "4                   Lehman Brothers   \n",
       "5                   Lehman Brothers   \n",
       "\n",
       "                                         eval_output  \n",
       "0  {\"reasoning\": \"The first list contains 'Americ...  \n",
       "1  {\"reasoning\": \"The entities in listA are 'Fede...  \n",
       "2  {\"reasoning\": \"The first entity in listA is 'M...  \n",
       "3  {\"reasoning\": \"ListA contains three entities: ...  \n",
       "4  {\"reasoning\": \"ListA contains 'Lehman Brothers...  \n",
       "5  {\"reasoning\": \"Both lists contain the exact sa...  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.select(t.response, t.ground_truth, t.eval_output).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a61994b-a489-4fc9-b71e-ec08ea9d2b85",
   "metadata": {},
   "source": [
    "Finally, it's time to pull the `decision` out of the JSON structs returned by OpenAI. There's just one complication: the `chat_completions` responses are strings, not JSON structs. We can resolve this with Pixeltable's handy `apply` method, which turns any Python function into a Pixeltable function. In this case, we'll apply the Python function `json.loads` to parse the string into a JSON struct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2283ef95-d344-4bae-a0e4-1a16bb190e9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing cells: 100%|███████████████████████████████████████████| 6/6 [00:00<00:00, 588.74 cells/s]\n",
      "Added 6 column values with 0 errors.\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "t['eval_decision'] = t.eval_output.apply(json.loads).decision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e6892c65-b104-4acd-93d8-242c70efbd76",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>response</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>eval_output</th>\n",
       "      <th>eval_decision</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Federal Reserve, AIG</td>\n",
       "      <td>American International Group</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;The entities in listA are &#x27;Federal Reserve&#x27; and &#x27;AIG&#x27;. The entity &#x27;AIG&#x27; stands for &#x27;American International Group&#x27;, so both lists refer to the same organization. However, &#x27;Federal Reserve&#x27; is not mentioned in listB. Therefore, since listB does not contain all the entities from listA, the lists do not match.&quot;, &quot;decision&quot;: 0}</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Merrill Lynch, Bank of America</td>\n",
       "      <td>Merill Lynch, Bank of America</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;The first entity in listA is &#x27;Merrill Lynch&#x27; while in listB it is &#x27;Merill Lynch&#x27;. The correct spelling is &#x27;Merrill Lynch&#x27; with a double &#x27;r&#x27;. Since there is a spelling error in listB, the two lists do not contain the same entities.&quot;, &quot;decision&quot;: 0}</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The company names extracted from the text are:\n",
       "1. Lehman\n",
       "2. JPMorgan Chase\n",
       "3. Lehman Brothers</td>\n",
       "      <td>JP Morgan Chase, Lehman Brothers</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;ListA contains three entities: &#x27;Lehman&#x27;, &#x27;JPMorgan Chase&#x27;, and &#x27;Lehman Brothers&#x27;. ListB has two entities: &#x27;JP Morgan Chase&#x27; and &#x27;Lehman Brothers&#x27;. Although &#x27;Lehman Brothers&#x27; is common between both lists, &#x27;JPMorgan Chase&#x27; from ListA is written as &#x27;JP Morgan Chase&#x27; in ListB, which indicates a discrepancy in the naming format. Therefore, the two lists do not contain the exact same entities.&quot;, &quot;decision&quot;: 0}</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>American International Group (AIG)</td>\n",
       "      <td>American International Group</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;The first list contains &#x27;American International Group (AIG)&#x27; which includes the full name along with its acronym, while the second list contains only &#x27;American International Group&#x27; without the acronym. Although both represent the same entity, the presence of &#x27;(AIG)&#x27; in listA makes them differ slightly in terms of representation. However, they refer to the same organization.&quot;, &quot;decision&quot;: 1}</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Lehman Brothers, Federal Reserve Bank</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;ListA contains &#x27;Lehman Brothers&#x27; and &#x27;Federal Reserve Bank&#x27;, while ListB contains only &#x27;Lehman Brothers&#x27;. Since ListB is missing &#x27;Federal Reserve Bank&#x27;, the two lists do not contain the same entities.&quot;, &quot;decision&quot;: 0}</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>{&quot;reasoning&quot;: &quot;Both lists contain the exact same entity &#x27;Lehman Brothers&#x27;. There are no differences in spelling, capitalization, or additional characters. Therefore, the lists match perfectly.&quot;, &quot;decision&quot;: 1}</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "                                            response  \\\n",
       "0                               Federal Reserve, AIG   \n",
       "1                     Merrill Lynch, Bank of America   \n",
       "2  The company names extracted from the text are:...   \n",
       "3                 American International Group (AIG)   \n",
       "4              Lehman Brothers, Federal Reserve Bank   \n",
       "5                                    Lehman Brothers   \n",
       "\n",
       "                       ground_truth  \\\n",
       "0      American International Group   \n",
       "1     Merill Lynch, Bank of America   \n",
       "2  JP Morgan Chase, Lehman Brothers   \n",
       "3      American International Group   \n",
       "4                   Lehman Brothers   \n",
       "5                   Lehman Brothers   \n",
       "\n",
       "                                         eval_output  eval_decision  \n",
       "0  {\"reasoning\": \"The entities in listA are 'Fede...              0  \n",
       "1  {\"reasoning\": \"The first entity in listA is 'M...              0  \n",
       "2  {\"reasoning\": \"ListA contains three entities: ...              0  \n",
       "3  {\"reasoning\": \"The first list contains 'Americ...              1  \n",
       "4  {\"reasoning\": \"ListA contains 'Lehman Brothers...              0  \n",
       "5  {\"reasoning\": \"Both lists contain the exact sa...              1  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.select(t.response, t.ground_truth, t.eval_output, t.eval_decision).show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
